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- ABSffBACT , ■ 

The lev York State Education Departaent conducts a 
Papir Eialaatiofl Program (PEP) in vhich each year, all-^third, sixth, 
and. ninth grade 'stadents in 4;he ^tate- are given a series of 
achievcnent tests in reading and aathe^atics. The= data acdaanlated by 
tjie departaent inclades achieveaeEt test scores, teacher 

--charactttistics,. building and curriculaa data, and census data. This 
extensive data is organized 50 that analyses hay be aade at the 

. building or district: level. , Using the data^^roa grades three and six, 
V factor analytic^studt of the achieveaent' data in -reading and 
nathenatics~is discussed- for each grade and the developaent of 
achleteaent criteria. Oslng the'se criteria and regression^nalysis, 
the variance aaong controllable and uncontrollable school 
Characteristics is partitioned to study the independent contribution 
-of specific variables of interest and nonlinear and interaction 
effects. (Authof/DEP) . 
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School Effects on Ac* leveoent^ 

Rjberf Illchols - !:<^»«>*o*o5;jM«fmTi^«,5r 
State University cf New York at Baffalo JSSiJJSJ^iStSrStSSrT?'^ 

71}c last decade has seen a revolution in our thinking about the effectiveness 
^ of schools-. Recall the optimistic euphoria in*thc early I960's nhich led to the 

rvi ^ . 

^ beglnnlna of the Heatf Start prpgrara and the passage* of the Elen^ntary and. Secondary 
BJucatlon Act of l965^-Ji^.ji^i*Mtion cowmjnity e^er and oonf idehtial ly took 
on such rajor responsibilities as breaking the cycle of" poverty and irestorliifg 
Mierlcan Intellectual leadership -In the world after the Russian sputnik. And 
seldom was head a discouraging word. - ' 

The first* discouraging word canje^ In 1966 with the publication of the Coleman ^ - 



Report and the dlsocMicerting finding that no amount of reanalysis would make 

'disappear. Hofne background was the najor correlate of student adhlevement and ^ 

when^this was controlled schools had Mttle adSitional effect. 

• jn 1969 Arffiur Jensen--began-_aiL^rt|cle with the st^teinent that a>f!ipensatory 

education had been tried and had fal led. He was mat with shocked disbel l.ef and 

inorb*! indlgnatJon. - ' ^ , ^ ' ^ 

In 1971 a group of -five Rand Corporation researchers ,ln an Influenclal review 

tor the President's Commissions. on School Flnancfe^ entitled- •'How Eff^tlVe As ^ 

\ ' ^ : * . " ^ * , . / ' 

School ing," concluded that no variant of "the existing jsystem has been shown to ^ , 

be consistently -related to students' educational outdbmes. They-wehT on to point 

out mat if, all schools are equaj ly effective but^re not equally exp^slve> 

IXv prudent man might reach a fairly obvious conclusion. 

CO Now* on the tenth anniversary of the publication of the Coleman report one 

. ^ can say without anticipating much argument that Individual differences In school 



^Paper presented as part of a syfnf>os\m. Searching for School Effects Through 

Conventional Multlvarfate Analysis and_ Through fbe/Study of School Outliers. 
American Educational Research Association Annual Meef Ing,^*^^'* franclsco, 
Calif., April 22, 1976/ 



adiievssent are doe priisarjly to heredity cr^to the early hocie envlronnieftt, so 

r 

%rhat the school does will inake little different ^ - - 

Personal ly, I beifevc half of this sfateoent* The evidence Is convincing 
that in the United States differences In Intel llgence and In school achievement 
are determined largely by events occurring prior to the onset of formal schooling 
and that most of these events occur prior to birth. However, I dcn*t think It - 

follows from this that schools have little effecK ifhat does foflow Is that- 

-— — _ * * ~ * 

the school effects are smal l relative to the-ef f ects of other factors, not that 
they are unic^rtant or uninteresting or of little value. If, for example, hone 
bacl^rounds and genes^are sonsehow hon^pgenlzed until they produced only small 
Individual differences, one might expect tJ:at schools would suddenly loom Into " 
view as a potent ancf varied source of Individual differences In -achlevaiT;3nt. 

The most beautiful muslfc that coirtes from the radio Is encoded as a relatively 
small signal on a powerful carrier wave. It seemS; probable that school effects 
are In a similar way superimposed as a smal I Inf luence-on'a very l^rge bacfgrouhd 
varl^tjcn^ snd-as a result tJieir detection and study pose very diff Icu I f 'research 

problems. At present we '^re at the crystal -set stage and what we hear Is mainly 

* * - - 

static. Stepwise multiple regression Is an ef fective crystal which wi l l' detect 

the school isignai from the background carrier wave, but we do not yat have ^ 

vacuum tube acipl lfier or effective fllters^ for extraneous noise, and we arc 

a long way from the superhete^^^ne circuit. 

Before nrientlonlng some possible amplifiers and filters let me first describe 

pur recent crystal-set study. Jt Is not materlafly dlffereot from other large 

school effects studies that have used the schobLas the unit of analysis. But 

perhaps the context for Interpreting the results is now different. We will no 

longer be astonished to find school effects that are minlscule In comparison 

with home background effects and that are gufte^ small relative even % the anblent 

noise level « This new perspective may •enable us to listen more attentively *f or 



i^he'inporfent csessas^ ^^^a^ J^a^^ ?very reason to believe Is there- ^ 

The analyses 1 will describe use rsuch of the same dafe' discussed in this 
syjiiposiua'by Suanson and Irvine- The Swanson analyses were done prior to those' 
i will descjribe and did not have access to census data. They- also ijltluded. 
private schools as vrell as publ ic schfcb Is. The Irvine analyses used the scIk)o1 
district rather than the school building as fhe unit. Thus, our ISree papers 
represent soaewhat different ways of locking at essentially the^sacie data:; 

The School Effects Study, , . , ' 2 

* 

The New York State Education Deparhnent conducts a Pupil Evaluation Program 
(PEP) In which each year all third and sixth grade students in the sta^e are 
given a series of achievement tests In rejadihg and math^atlcs.? Student scores 
o» these tests were aggregated for school buildings^ which yielded surnrary 
jaeasures for five subtests for the two grade levels fo/^ the years I969*thrbu3h 
1972, A series of factor analyses of these data, using a sample of 829 schools 
revealed a large general factor accounting fori about 80 percent of the variance 
among al l the variables* involving mean performance. ,Thus, there was a very 
definite tendency for schools to score generally high or generally low over bgth 
grades ^for all four years on al l five subtests, -There wa^^^^thus; ample justif I- 
cation for combining the means for the five subtests Into a single^,General achieve 
ment* score for each grade leveJ for the 1972 tests for use as the achievement 
criterion for this study. 

' The factor analyses also revealed smaller factors .1 dent If led with such 
dimensions as mathematics vs. reading test consent,, grade level, within school 
Variability agd s^ewness of score distributions and temporal change over four 
years. We have done studies of these other ^character rst I cs=of test performance 



^The ninth grade Is also Included In PEP, but was not used Jn this study bec5^use 
not a|l ninth grade students were tested during the years studied. 



ifj a school , tat will not report them here other than to say that correlations 
of school charactei I:>tics n?ith these criteria were s^inerally isuch, lower thjn 
for general achievement and usually were not very fnfonriative. In this study 
our interest is in the general ^achieve-nent level of the school* 

A variety of additional infornstion was available abou^ the sci)ooIs pr ^ 

* 2 

*their district for study, in relation to the achievement critericn. These 
variables laay be considered as fall iny into two broad cat<^orJes: 1) Input 

variables which describe the population of students attending the schcoj ar^T^ 

» 

treatrjent variables which relate to the educational prograni of the school* 
The input variables were obtained f rem ^three* sources • 

1. Reports of characteristics of the students in the school by a school 
official to the Sieto Education Department's Basic Educational Data Systan (BEDS)* 

2. ihe geographical locat'ion of the school. * 

3. Data irom the 1970 U.S. census concerning 4-he school distcict in which 
the school is located. Qensus data have been aggr^^ated for the 734 School 
districts in New York St^ste, and^data for ihe district v;as applied to all schools 
in that district* Although New York City is one^schpof district, census data 
were aggregated separately for 31 sub-'districts. in that city. ' ^ ' 

The treatment variables were of three general types af I obtained from the 
Basic Educational Data System. - 1 ' - ^ 

1. Characteristics of the educational program 'of the school. 

* \ ' * 

2. Characteristics of the teachers in the school, aggregated from Individual 

*' ^ 
reports to tjie State Education Depari7ner\t by all teachers In the state. 

3. Financial resources avai lable ta the school district In which the school 

Is located. \ • 

" ' • \ 

The specif Ic variables availably in these categories are listed In Table ). \ 



L 

^ "Tho data were analyzed separately fcr 1701 third-grade sdicx>!s and 1S33 
sixth snade schools-- AInast all' of the sixth grade schools* were also In the " 

third grade sample, but about 300 schools had a third grade and no sixth grad(^ 

z * . * 

This is essentially all the third- and sixth grade public schools in Kew York 

State- Only a fe>^. stfjre excluded because 'iiripbrtant data were missing. Al though 

the^PEP tests are also given in private school s^ they were not used in this study 

bec^se sone of the other variables i^ere not ava'iJefcle for theni* ^ 

, The ZQTO-orA^r correlations of the four school achievernant measures with 
tbe^4 school input variables and with the 24 school treatinent variables are ^ 
shown' in Table 1- This table shews that the correlations of the Input variables 
with achleven3aht tended to be Xqtq^v than were those of the treatrsent variables 
§rs Ke have locrned to expect* ^ - 

. The largest input correlations were with Indices*6f the economic status of 
the students' home* A school official's report of the proportion of children 
in the school from families on welfare correl3ted -.74 and -.76 wifr. mean cchieve 
ment. The <x)rreIations with thg two census. Indices of the Incidence of poverty 
v;ere also substantial , but about ten points lov/er, probably because .the cdftsus 
data apply to, the. school distrlcrf rather than to the individual school and they 
. were obtained tivo years earlier. The correlations with mean fami ly Income v/.ere ^ 

still lower suggosttnj tha^^ Incidence of poverty is a more^salient factcr 
0ffor achievement than I5 the general level T^f^affhience.' 

:^ This finding of a negative relationship betv/een poverty and achievement Is 
certainly not new, but the very high correlations observed on a state-wide basis 
should make us pause to consider the remarkaWe degree to which low. achieving 
schools are located»ln poverty areas. Consider that 75 Is about the test- 
retest rel iabij ity of our better personality Inventories. This cprrelatiQn is 
larger than those often reported for thesQ variables because It Is based on 



schooi-fDsans rathar ^an on individual students. Yet when we }ook at schools 

* » " i 

as tte*jjnit this is^what we see. As one moves frocn the ;fop.*fJvp percent or the 

schools Kho report no poverty students to the lower five percent of the schools 

who report Dore than 70 'percent poverty students average achieveiaent goes fron) k 

one-and-a~ha I f standard deviations above the^fsean to one-and-a-half below* 

. Table 1 & J s^ shows a correlation of about\6 between the achievement • I eveV 
* » 

of the school &nd the educational level of the j>opulation of the district in • 
which , the -sc too I is located/ This cxjrrelairion was cut almost in half by stafis- 
tical control for the economic variables, but it still remained a substantial 
and highly signif icant independent contributer to the achievement level of the 
schools - ^ " , * • ' 

The high negative zero-order correlations of percent^e Itegrp and percentage 
Spanish American* students with scfyool achieverrsent that are shown in Tablg 1 were 
reduced to betv/een -'.2 and -.3 by contrpK for the epDnomlc and eaucaflcnal ' 
variables, and most of thJ's reduction was due to, the .econoniic variables. But 
with these controls the racial and ethnic correlations were "still highly si'gnlfi- 
cant and were the third most important set of. contributions to achieveinent after 
the economic and educational variables* 

After control for the^onomic, educational and racial-ethnic* variables, no 
other4nput_yanable had a ,partlar^rrf:lation with achjeya^ent^s ihigh as . I . 
Although several were statistically sjjgntilcant because of the large sample, 
these were not consistent across the two grade levels. 

, One additional background variable/Jeserves corornent. If only because New' 
York City has had more than Its share of bad press recently. The correlatron 
In Table 1 -of about -.3 of l^he New Y^k City dummy variable with'achievemsnt 
Shows that low achieving schools tend to be concentrated In the city. However, 
after^ control for economic, radal^ethnlc and educational variables thi^ correla- 
tion was reversed in sign and was significantly positive for third grade schools. 
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Thus^ the relatively low ecnievenent Jevel Fn Mew York City can be attributed 

to sociai factors that are already welJ known and does not represent the discovery 

' of. 3 ffew problcfn for'the beleagjred cityi Alas, the same reversal did not hold 

for the upslfete center cities. * 1 \ 

\ The fDuItipJe correlation with*school achievernejft of al l input variables 

that contributed significantly to the equation was .64 for third grade and .85 

for sixth grade. - Tiiese multiple correlations are shown In Table 2. Thus> we 

can account for a whopping 70 percent of the variance In' mean achleviOTent level 

of school buildings on the- basis of the background characteristics of students 

In the school, primarily the economic, educational and racial-ethnic status 

of the students '^<ftomes. " ^ * ^ 

• Vte now move to^the* relationship of mean achievement to the school treatmerft 

variables. The zero-order correlations In Table ! show that the largest re- • 

loMonship was with ojDerating expenditure per pupil, followed by a group of 

related variables reflecting the a&e, experience and salary of^the teachers In 

^- ^ ^ - » . 

the school. However, these school characteristics were also correlated with ' 

^_ - ' i 
the, ecQnomIc educational and racial-el^nfc predictors of achievement. '.It is, 

\ 

thus,- necessary to control these bapkgrbjund. factors to study more directly the 
effects^of school variables on achievement'.'' - ^ - 

OuV crystal for detecting small school effects In the presence of large 

background variation is st^p-wlse multiple regression analysis. We al lc^/ the 

— " ' ' ^ - * \ ^ 

"program to bul-ld up a regressjon equation by adding_ Input variables one at a time 
unti l nc^ additional Input variable contributes signif icantly to prediction of 
achievement Essential ly the same results are obtalajed/by forcing al f Input , 
variables Into regression at one step, but the flnaF equation Is Somewhat neater 
with the stepwise procedore and the^ Individual stejjs^provlde additional I nterestti ng* 
Information. V/e then look at the partial correlations otJ:be-.treatment variables 



with echievesnent contralling f^^^S^ignlf icarit i nput j>red ? cto rs >^ At this 
pa-int «e have a cho}^^<Sffhree coefficients to represent these residual 

1. The beta ifeight the treaiSnenf variable would jia^ if added to the equation 
vith the input variables-' This Is the relationship of the treatment variable 

.as is to The criterion as is Independent of all the input variables. These are 
the path cosff Icientis us'ed iii^^th anajVej s , ^ v 

2. The part correlation. This, Js the relationship between that part of ♦ 
the* treatJT^nt vorlable that is Independent of the input variables vMth the 
criterion as is, - . . . ^ / 

3. The partial correlation. This is the relationship between two sets of 

residuals with the inout variance taken out of both the treatment variable* and . 

>\ ^ * 

* - * ' - , , 

the criterion. - . . - - *^ 

^ » ^ 

V/hen the input variabljes accounts for a large amount of variance, as they 

V 

usually do In school effect studies, the difference betjveen ^these three coefficients 

— " * ^ » * 

is substantial. In the present analysis the beta welahts and the partial correl- a 
atlons differed by as much as a factor of three or four. Thus, If one wishes to 

J ' ' , 

make the school e#fects look smal l he should use,- the partial, correlation. If he' 
wanis them to look miniscule he should use fife beta v/elght. The part correlation, 
is an intermediate choice- ' .v . - ^ ' 

- ' Since our school^ effects need a 1 1 the help: they can ^et, v/e chose the ' *\ ' 
partial correlation to represent them In Tgble 3. JThese coefficients, thus, 
represent the correjatlon to be e^^pected betv/een treatment and achievement 
variables witfv the greatly reduced variance they v/ould have if the Input 



variables v/ere he Id* constant- 



i 



Table 3 shows that operating expenditure per pupl>, .which had a la^ge zero- 
order correlation wllh achievement^ v/as not significantly related after control <^ 



for student inputs the cluster of 'varfablos representing teacher experience, v 
,age and-sal&ry, however, remained significantly relatecJ to achieyemeot in both 
tKi rd anb -sixth grade. * ' '* . 

In addition, academic orientation of the school as might be "Inferred' from 
traSitional classroom arrangement and^use of rooms for academic purposes w|re 
signi:f4cantly* related to achievemont after control for student input. 



The significant negative partiaJ correlation of the presence of compensatory 
programs v/ith achievement proLably ref lects the fact that these programs tend to 
exlsjf* where they are needed to an extent that was not completely washed out by 
our Input controls^ 'At leasj* this seems a more reasonable interpretation than. 
attributing some detrlmi^ntal Influence to such programs. However, there is a 
cost-,of making such a tel Icitous ipterpl;e^ation• ^To do so,, we must admit that 
ouf Input controls leave something to be desired, and, tp be consistent, iye must 
extend the saftie rnterpretatipn to fhi positive partial, correlation of achievement 
with the- presence^of programs for the academical ly talented. 

This suggestion of incomplete jodht'ro I fpr Input mjght be dlsmi^ssed as ^ . 
Jnconsequentiaf If the partial correlations with other treatment vgr i ab 1 es were , 
relatively large, but they v/ere not. The largest possible effects accounted 
for only, pne or two 'percent ^6f the residua P variance. In achjevement. Thus, the 
problem of input control Is ser[ouSt Since.^ Input accounts tor about 70. perceht ^- 

the total variance In achievement, controls for Input that v/ere as much as 
90 -percent effective could still leave uncontrol led input effects about as large 
as the^scjlooi effects that are likely to te observed. It Is. extremely Impprtant^ 
to control as much of the Input variance as possible, but It seerns Inevitable 
't^-iat ^ background noise will remain that Is almosf as loud as the slgnairt Like 
5»the^arly crystal se^ l isteners we must strain our ear to> separate the mean Ingful 

; . - . --v . 

signal from the annoying statl,c« ^ * 

• . ,10 ■ - • ■ ■ ;■ ... \ 



I 
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The finding that the cl uster of i^elated teacher variables ~ experience^ ^ 

pee, edqcatlon and salary ^'was positively related to achievement after input _ 
' ■ - ■ . • ^ ' . . - • ■ / 

controls at a high level of slghl:f i.cance at both gra(^ I eve I s seems to me to be ^ 

' a meaningfu.1 signal that stands out from the noised Similar fln.d'ing.s- emerged - 

from the analyses reported In this symposium bySwanson and by Trvirie. Although ^ 

these ana'lyses-.were based on essfentiajriy .the' same data tlhe approaches were 

different, .tbe input^output .studie? covered by th^' review by the Rgnd Corpdrat/on 

researchers frequently repor*ted ppsitlve effects for teacher experience and ^ 

" raVe I y reported hegatjve effects. . " . 

^ Perlpaps all of this is*beg inning to form a meanlrigful pattern, but the'^ 
i ' " \ 

pattern Is represented I'ii a'signaJ so weak that If Ts difficult to perceive. 

What Is needed Is ah ampldrfl'er. '.The extreme groups appro|£h .used by Swansoh ^ 

very effect I veLy amplifies the iindings,^ but It ai.s<5 amplifies, the noise-so we ^ 

■ ' ' • - . , , t • - . - • 

are not -much "better off. However, the extreme groups approach facilitates 

an- additional step, as Swansoh suggests, which v/i 11 amphify th^ signa I and -not 
• '■ ' - * * 

the noise.*- It Is not the age or the exp^ience-pr the. salary of the teacher^ 

' • ' ' ^ ' ' ■ . ' ' 

In a.school.^tWt promc^tes stgdent achievement.* It is something that impenges 

. - \ ■ ■ i ' * ' 

mucV-igpre 'dlreqtly oh the students sr.d that is very imperfectly represented 
by. these- gross, indicators. "."The results so far suggest hypotheses about what 

' . ' i • .' 

■ the jiore sa p ent ' va.r'iab les a re \ \ ke I y, to be V The out 11 er approa.ch ma kes ' i t ^ ' ' ." 
"feas'lbTe to co.l lect additldrtal data frpm extreme s'chool Wo, test.thele hypotheses. 
I n: feojic I us i on, h-oge you w 1 1 1 pet^i t. me" to- car,rV '♦'he rad io ana.Logy one 

» step *furthep and suggest that the co'ntrdll ed exp(?ri|Tient is an effe6tive f i lter,. 

\ . ■* — ■ • ' J ' ■ • 

which selectively enhances the'slgnal" and, attenuates the noise.' Yet it regret- 



ably has seriqusJ_LmItatIons - for the stiidy-pf: school effects In the resrl world. , . 
It l/expenslve. It Is pal itical ly dlfffcu.lt. to" implement,- and It must be pre- 
cisel.y turned^to thfrspeclfic effects to be .detected. Tliu^,- the most appropri^^te ^ 
research strategy a.|>.;p resent seems' to be to -cpnti n'ue. the broacl-b^hd-wIdth'' b.ut 
static-bidden regr(^ss]on and outhler. s.tudjes untl P strong fhferences are. ^ H 

" . r / ' H ^ 

iyalj Wie. that^re wort-hyiof expe'rlmental- manjpuJ ' ^ 
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Partial Correlations of School Treatsnent Variables 
CcSntroii ing for School' input Variables 



I 



/ 



iSchool Characteristic 



Partial Cbrrelafion 

Grade o 
CM - 1453) 



Grade 3 
(N = 1701) 



Characteristics of- School PTOprans 

•Student/Teacher rafio ' 
Prqgrans for the academicaNy "talented 
Con^erisatory progran 
* Students/total roonis ratio. 

Students/regular classroorr£. ratio 
ft of rporas used for academic purposes. 
Classroom arrangement tradi.tional 
Classroora arrangesnent cluster 
Classroo-Tj arrangement open 
C6rre£rf"ive reading program 
CorreCTive speech program 

Characteristics of Teachers jn the School 

Uncertified 
% wi+h le^s than Bachelors degree 
% Sachelors degree only 
% w.ith I<5asrers + 50 hours or Cocforate- 
i Occupation other than teaching last year 

Tenured 

^Full-time . * 
Mean degree status 

f'fean number of years experience in this district 
I'tean number of years teaching experience 
Kean:.sarary 
ftean age . 

FinanciaT Characteristics of School District 
Per pupi I expenditure 



.03^ 
.05; 

-.07 
^1 

-'.05, 
-05 
.04 

-.0! 
-04 
-01 
.00 



-.03 

--06 
--01, 
-.05, 
.07 
.03 
.04, 
.09^ 
,.08, 
.08, 
.09 



-.02 



-04, 

.11 
--02 

.01^. 
-.04, 

.07, 

-05 
-'.01 

.01 

.04 

i03 



.00, 

.06, 
-.09 

.02 
-.03, 

.10 

.06, 
-12, 

.09, , 
.13 



.00 



p <.05 
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